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Abstract: A controller-in-the-loop simulation was conducted in the Airspace Operations Laboratory (AOL) at the NASA 

Ames Research Center to investigate the functional allocation aspects associated with ground-based automated separation 
assurance in a far-term NextGen environment. In this concept, ground-based automation handled the detection and 
resolution of strategic and tactical conflicts and alerted the controller to deferred situations. The controller was responsible 
for monitoring the automation and managing situations by exception. This was done in conditions both with and without 
arrival time constraints across two levels of traffic density. Results showed that although workload increased with an 
increase in traffic density, it was still manageable in most situations. The number of conflicts increased similarly with a 
related increase in the issuance of resolution clearances. Although over 99% of conflicts were resolved, operational errors 
did occur but were tied to local sector complexities. Feedback from the participants revealed that they thought they 
maintained reasonable situation awareness in this environment, felt that operations were highly acceptable at the lower 
traffic density level but were less so as it increased, and felt overall that the concept as it was introduced here was a positive 
step forward to accommodating the more complex environment envisioned as part of NextGen. 
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1. INTRODUCTION 

In the United States’ National Airspace System (NAS) of 
today, air traffic controllers manage operations within their 
sector of responsibility through active involvement with 
each aircraft that enters and exits. This allows the 
controller to incorporate each aircraft into their overall 
operational picture and has facilitated the current levels of 
safety. However, the airspace envisioned for the Next 
Generation Air Transportation System (NextGen) is 
expected to be both higher in traffic density and 
complexity relative to what is experienced today (Fig. 1 
presents an example of how such an environment would be 
displayed without changes to the current system) [1]. The 
transition to this future system will certainly require 
changes in a number of areas, without which operations 
within the NAS would likely become unviable. 

Given that the primary responsibility of air traffic 
controllers is to maintain safe separation between aircraft, 
the area of separation assurance (SA) has been identified 
as one requiring a concerted and focused research effort 
with advances in this domain serving as enablers for the 
transition to NextGen. Within this framework, the thrust of 
some of the recent research has been on addressing the 
impact of higher levels of traffic density and complexity on 



the workload of air traffic controllers -a potential barrier 
to NextGen realization- and exploring the proper levels of 
automated support necessary for the accommodation of 
increased traffic and complexity without creating excessive 
workload and compromising safety. 


This paper will first outline the concept of operations that 
some of the research has been predicated upon followed by 
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brief descriptions of previous SA research conducted in the 
Airspace Operations Laboratory (AOL) [2] at the NASA 
Ames Research Center. This will then be followed by the 
presentation of the most recent research effort with 
accompanying results and will conclude with a discussion 
of those results. 

2. GROUND-BASED AUTOMATED SEPARATION 
ASSURANCE 

In today’s environment, separation assurance is maintained 
through a largely manual process where the air traffic 
controller scans and identifies potential separation risks, 
develops a conflict avoidance resolution based on their 
understanding of the current operational picture, and issues 
a verbal clearance to the appropriate aircraft. While this 
process has been and is carried out very effectively, it will 
become an increasingly difficult task if and when traffic 
demand in the United States reaches the increased levels 
forecast by the Federal Aviation Administration [3]. 

One of the changes that would provide for the ability to 
accommodate such levels of traffic is the introduction of 
greater levels of automation in support of the separation 
assurance function. Assuming an environment where 
trajectory based operations (TBO) are being conducted, 
such automation would perform the duties of conflict 
detection and develop resolutions for nominal conflict 
situations and uplink conflict-free trajectories to the 
appropriate aircraft via data communication channels. 
Tactical conflict situations with an imminent loss of 
separation would be handled by an additional layer of 
automation where tactical vectors and/or altitude changes 
would be sent to the appropriate aircraft via a separate 
communications channel dedicated to such transmissions. 

Such changes to the task of separation assurance and the 
involvement of such levels of automation would signal a 
shift away from an individual air traffic controller having 
the sole responsibility for maintaining safe separation to 
one in which the responsibility rests with the service 
provider- a term used to refer to the collaborative team 
composed of the air traffic controller and supporting 
automation. This shift would not only allow for increased 
airspace capacity, but would also allow for the air traffic 
controller to provide additional services while managing 
the airspace by exception. 

3. PREVIOUSSA RESEARCH IN THE AOL 

The components of the larger concept of ground-based 
automated separation assurance as well as the concept 
itself have previously been detailed and presented as part 
of the Advanced Airspace Concept (AAC) [4-5]. Early 
simulations and analyses in this particular area were 
conducted through fast-time simulations and modeling, but 
investigations into the operational and human factors 
issues surrounding such changes as envisioned in the 


concept as well as the environment in which it would be 
realized were limited. 

To date there have been three controller-in-the-loop 
simulations conducted in the AOL specifically related to 
separation assurance. Each of these has built upon the 
research and results of the previous study and has served to 
inform the next while incrementally introducing greater 
levels of automation. For simplicity these studies are 
referred to as SA1, SA2, and SA3 according to their 
temporal order. 

3.1 SA1 

The first in this series, referred to as SA1, was conducted 
in 2007 and evaluated the differences in performance 
between participants resolving conflicts in a manual and 
interactive mode as well as a fully automated mode [6-7]. 
This was done at current day (lx), twice (2x), and three 
times (3x) that level of traffic. At the heart of the 
interactive and hilly automated modes lay the autoresolver 
algorithm outlined as part of the Advanced Airspace 
Concept. Workload impact and acceptability of the 
algorithm’s resolutions were also investigated. Results 
suggested that the automation provided significant benefits 
in terms of safety and efficiency particularly at higher 
levels of traffic. There was also a significant reduction in 
workload. The resolutions provided by the automation 
were also rated as being generally acceptable. 

3.2 SA2 

The second study, SA2, was conducted in 2008 and tested 
air/ground operations in an environment where ground- 
based automation was responsible for safely managing 
aircraft trajectories [8]. Controller participants were 
responsible for handling pilot requests that were deferred 
by the automation and also handled scripted off-nominal 
events and tactical conflict situations both with and without 
Tactical Separation Assurance Flight Environment 
(TSAFE) support [9]. Operations were conducted at 2x 
and 3x levels of traffic. Results showed that the strategic 
conflict resolution automation was able to resolve 98% of 
conflicts, 95% of uplinked trajectories were rated as 
acceptable to the flight crew participants, and workload 
was generally low. Of the tactical conflicts, 75% were 
resolved, which served as a springboard for discussion of 
the issues related to such automation and how they could 
be addressed in future research [10]. 

3.3 Current Study (SA3) 

The third study in this series, SA3, was conducted in part 
to begin addressing the concerns expressed by the Joint 
Planning and Development Office (JPDO) regarding the 
“lack of clarity” surrounding the functional allocation of 
new functions and responsibilities between the ground- 
based air traffic control systems and the flight deck- based 
systems [11]. To that end, two separate but collaborative 
studies were conducted with respect to the functional 


2 



ENRI Int. Workshop on ATM/CNS. Tokyo, Japan. (EIWAC 2010) 


allocation of separation assurance with scheduling 
constraints between the ground and the air, with the 
ground-based focus residing at the NASA Ames Research 
Center and the flight deck-based focus residing at the 
NASA Langley research center. Results and discussions of 
the collaborative aspect of this overall study have been 
published previously [12]. This paper will focus 
specifically on the ground-based portion of the study with 
a further examination of the allocation of separation 
assurance functions between the ground-based automation 
and the human air traffic controller. 

4. METHOD 

This study was conducted over the course of eight days in 
February and March of 2010 in the AOL at the NASA 
Ames Research Center. Training and familiarization was 
performed for the first three days with the remaining five 
days being devoted to data collection. 

This section will begin with an overview of the experiment 
design and operational environment as it was presented in 
the study, followed by a description of the apparatus, 
participants, airspace, and the procedures used during the 
data collection phase of the study. 

4.1 Design 

The overall ground-based study included three different 
components. The first consisted of short length (S), 15- 
minute runs that involved three different types of 
scheduling constraints (see Fig. 2). The second consisted 
of medium length (M), 30-minute runs that involved 
scheduling or no scheduling across two different traffic 
densities referred to as NextGen Levels A and B (see Fig. 
3). NextGen Level A was roughly equivalent to 1.5 times 
current day traffic levels and NextGen Level B was 
roughly equivalent to 2.0 times. The S and M runs were 
used to satisfy the collaborative requirements of the larger 
functional allocation study. The third component of the 
study was an independent aspect used in a more 
exploratory context that involved long, three hour runs 
with scheduling and weather constraints and traffic density 
levels that fluctuated to the higher extremes. The 
procedures and results presented in this paper will focus 
mainly on the medium length runs. 

Basic (STA) 

Timing of Dispersed 

Arrival Time (one every minute) 
changes Synchronous 

(all at 6 or 8 minutes) 

Figure 2 Design matrix for the short duration runs 
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Constraints 
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Figure 3 Design matrix for the medium duration runs 

4.2 Operational Environment 

For this study, the operational environment was aligned 
with what is envisioned for NextGen where there is an 
integration of the flight environment and avionics, ground- 
based automation, and the controller workstation. 

Flight Environment 

The flight environment assumed full air-ground data 
communication and Automatic Dependent Surveillance- 
Broadcast (ADS-B) equipage for all aircraft transiting the 
airspace. Trajectory Based Operations were being 
conducted where it was assumed that all flights were on 
their user-preferred, 4-D trajectories and were cleared for 
their departure and arrival transitioning phases of flight. 

Ground-Based A utomation 

The instantiation of ground-based automated separation 
assurance for this study involved a number of related 
components working in sequence. The first provided the 
detection and reporting of conflicts with a 12 minute look- 
ahead time, followed by the development of conflict 
resolutions that were calculated to minimize delay, and 
finally the uplinking of trajectories to the appropriate 
aircraft without the need for controller involvement in 
nominal situations. Successful completion of this sequence 
relied upon accurate speed and position information. 
However, trajectory uncertainties and inaccuracies did 
exist as they do today and most likely will in the future. 
Conformance monitoring of flight trajectories detected 
when an aircraft had deviated from its path outside of 
acceptable tolerances, at which point the particular flight 
was alerted to the controller through an enlarged target 
symbol and a tactical, dead reckoning conflict probe was 
activated with a five minute look ahead. In cases where a 
conflict was detected late, i.e., with less than three minutes 
to loss of separation, a tactical vector was computed for 
one or both aircraft in the conflict pair and uplinked when 
the time to loss of separation fell below two minutes. This 
resulted in aircraft receiving the uplink to vector according 
to the uplinked trajectory, which resulted in the aircraft 
becoming off-track. It was then the controllers’ 
responsibility to put the off-track aircraft back onto their 
preferred trajectory in a safe manner. 

Controller Workstation 

The workstations and displays used for the sector 
controllers and supervisors incorporated advances in the 
display of information and access to functionality deemed 
necessary for control operations in a NextGen environment. 
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Fig. 4 presents a screenshot of an R-side’s display as it was 
implemented in the study where it can be seen that it 
differed significantly from what one would find today and 
as shown in Fig. 1. 



Figure 4 Advanced controller display at high levels of traffic density 


where scheduling and weather avoidance tasks were being performed 

The first difference one may notice in Fig. 4 is that the 
target symbols were chevrons with an associated limited 
data block, which displayed the current altitude. This is a 
necessary departure from today’s display due to the 
problem of data block clutter and overlap once aircraft 
counts in a sector reach levels predicted for NextGen. In 
this example, there are nearly 50 aircraft in the sector. This 
would result in a controller using today’s display to spend 
most, if not all, of their time simply managing data block 
positions. What enables this reduction in displayed 
information is the shift described earlier away from active 
control of each aircraft as it is done today, to one in which 
the automation manages the traffic and separation 
functions while the controller monitors and manages the 
automation. In this case, the information available through 
a data block is accessed on an as-needed basis rather than 
having the requirement of being displayed constantly while 
the aircraft is within the sector’s boundaries or under the 
sector’s control. 

Additional information and interactive functionality were 
integrated into the workstation to assist in the performance 
and monitoring of certain tasks. For example, on the right 
hand side of Fig. 4 one can see a timeline displayed that 
controllers used to manage the scheduling constraints in 
effect for area airports. This timeline featured the 
scheduled time of arrival on one side and the estimated 
time of arrival for aircraft scheduled to a particular airport 
or fix on the other. This served as an indication of how 


well the scheduling was being adhered to. In the event that 
an aircraft began falling out of conformance, the timeline 
provided an interactive function that allowed the controller 
to call upon an automated algorithm that would attempt to 
find the optimal, conflict-free solution through speeds 
and/or vectors to meet the scheduling constraint. 

Another item in the display that assisted in the monitoring 
and management task of the controller was the conflict list 
seen near the upper right hand comer of Fig. 4. While the 
ground-based automation handled the nominal conflict 
detection and resolution tasks independent of the controller 
in this study, conflicts relevant to the sector were displayed 
in this list with additional information regarding the 
vertical profiles of the involved aircraft, predicted time to 
loss of separation, closest point of approach, as well as the 
automation’s status and progress in resolving the conflict. 
Constraints on the automation were put in place that 
prevented heading changes of more than 31 degrees or 
altitude changes of more than 670 m (2200 ft) from being 
sent directly to aircraft without controller involvement. In 
such cases, the resolution of the conflict was alerted to the 
controller through the conflict list, at which point they 
would develop a resolution either on their own or by 
reactivating the conflict resolution algorithm. 

In both the scheduling and conflict resolution cases, the 
controller had access through the workstation and display 
to data communication capabilities. Once an acceptable 
solution to a given situation was arrived at, the controller 
had the option to type an uplink command into the 
computer readout device (CRD) or access a fly-out menu 
either through the aircraft’s data block or the callsigns in 
the respective lists where the uplink command could be 
activated. This then sent the modified trajectory directly to 
the aircraft. Similar to this functionality was one that was 
used for cases that required coordination rather than direct 
contact. In these cases, rather than selecting the uplink 
command, the controller selected the coordinate clearance 
command. This sent a proposed trajectory to the sector that 
had control of the relevant aircraft for their approval. If the 
proposal was acceptable, the receiving controller could 
then uplink the clearance directly to the aircraft. 

4.3 Apparatus 

The operational environment just described was simulated 
using the Multi Aircraft Control System (MACS) software 
package [13]. This is a Java based simulation platform 
developed and maintained by the AOL software team. 
MACS is a scalable platform used for the prototyping of 
displays and concepts that range from current day to the 
exploratory far term timeframe. 

MACS was the common thread shared between all of the 
stations and positions used throughout the simulation. This 
included radar controller workstations (see Fig. 5) which 
were composed of standard PCs with 75cm Barco monitors 
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and Display System Replacement (DSR) keyboards and 
trackballs as input devices. These workstations were also 
equipped with tablet PCs which were used for voice 
communications similar to the Voice Communications 
System (VCS) used currently. Area supervisors had access 
to two separate stations, each one with a different purpose. 
One station projected a Traffic Situation Display of the test 
airspace with accompanying traffic load statuses and 
predictions for the test sectors. The other station was 
similar to the radar controller station but was configured 
differently to enable the performance of strategic planning 
functions. Four confederate “Ghost” controller stations 
were used for the control of air traffic outside of the test 
airspace, which consisted of standard PC setups with 76cm 
monitors. Ten pseudopilot stations with standard PC setups 
were also used for the management of individual flights in 
the problems. Two simulation manager stations were also 
used for the initiation and termination of each training and 
data collection run. 



Figure 5 Radar controller workstations with TSD and test sector load 
predictions projected onto the wall above 


4.4 Participants 

Six current FAA front line managers, each from different 
enroute centers and current on radar, served as test 
participants for this study. Each rotated through radar and 
area supervisor positions in order to operate and evaluate 
the concept from the two different perspectives. Four 
recently retired controllers worked the remaining test 
positions. Four additional retired controllers acted as 
confederate “Ghost” controllers that managed the air 
traffic outside of the test area. Ten general aviation pilots 
acted as pseudopilots and were assigned to each of the test 
sectors and the surrounding areas. 

4.5 Airspace 

The airspace used in this study consisted of test sectors 
from the eastern portion of Kansas City Center (ZKC) and 
the western portion of Indianapolis Center (ZID) in the 
central region of the United States. This was high altitude 


enroute airspace with the floors of the sectors set at flight 
level (FL) 290. For the medium length runs, four test 
sectors were used: ZKC 90, ZKC98, ZID81, and ZID80. 
As seen in Fig. 6, the design of each sector is slightly 
different and the traffic flows and characteristics produced 
differences in density and complexity between them. 
Arrivals and departures from local area airports 
contributed to this complexity for some of the sectors with 
transitioning aircraft interacting with the level flights 
transiting the airspace. Confederate controllers handled the 
traffic outside of the four test sectors as well as the aircraft 
below FL290. 



Figure 6 High altitude test airspace from the central United States used 
during the study 


4.6 Procedure 

The first three days of the simulation were dedicated to 
training the participants on the tools, airspace, and 
concepts that they would be presented during the data 
collection phase. This was done through a combination of 
briefing discussions, overviews and presentations in the 
laboratory, and hands-on interaction with the workstations 
and software. The hands-on portion of the training was 
conducted by using traffic scenarios of varying levels of 
traffic density ranging from very light to the full scale 
densities that they would experience during the formal 
simulation. 

The traffic scenarios used in the short and medium runs 
were initially developed by the team in the ATOL as were 
the formulations of the ST A assignments that would be 
issued to aircraft in the appropriate runs. These traffic 
scenarios were then collaboratively adjusted and modified 
to ensure proper, uniform characteristics and performance 
in each laboratory’s respective platforms. In the AOL, it 
was also necessary to convert the traffic scenarios into the 
format appropriate for use in MACS. Python scripts were 
also developed for use in the issuing of ST A assignments 
in the runs involving scheduling constraints. Traffic 
scenarios as well as convective weather representations 
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were also developed in the AOL for use in the exploratory 
runs. 

The data collection runs were conducted over the five days 
following initial training. Each day was divided into three 
different phases with the mornings devoted to the 
collaborative short and medium runs. The first two runs of 
the day involved the thirty minute scenarios with various 
scheduling and traffic density conditions. These were then 
followed by three, short 15 -minute runs with varying ST A 
assignment conditions. 

These morning runs were conducted in parallel in two 
separate but adjoining rooms. Each room was equipped 
with four radar controller stations-one for each test sector- 
and two area supervisor stations to be used by a single 
supervisor. For consistency, each room was running the 
same problems in the same conditions but was completely 
isolated from one another. Within each room, the area 
supervisor could monitor the traffic situation by walking 
the floor and by assessing the flows and traffic load 
statuses and predictions for each of the four test sectors via 
the two supervisor stations. The four radar controllers, 
however, were separated by partition (see Fig. 5) with the 
two ZKC test sectors on one side and the two ZID sectors 
on the other. Inter-center communications and 
coordination were required to be carried out via voice 
channels and/or ground-to-ground data comm. 

During the runs, the tasks of the control team were 
different in many respects from what they are today. As 
this was a functional allocation study of ground-based 
automated separation assurance, the main departures from 
today’s roles and responsibilities were in this area. Table 1 
provides a delineation of the tasks performed by the 
automation and the human controllers during the data 
collection runs. 

For the medium duration runs, there were four conditions 
in which participants operated. In the two Baseline 
conditions, participants managed the automation and 


traffic situations according to the allocation of functions in 
Table 1. This was done at two different traffic density 
levels, which were approximately 1.5 and 2.0 times greater 
than current day (from here called Fevel A and B). The 
two ST A conditions involved the participants operating in 
the same manner as in the Baseline conditions, but with the 
added responsibility of maintaining the scheduled times of 
arrivals for aircraft arriving at a number of area airports. 
Automated support for this task was provided through 
timelines that were configured and assigned to each radar 
controller workstation. 

Data were collected from a variety of sources throughout 
the study for later consolidation and analyses. During each 
run, screen recordings were taken on each of the 
workstations. Every action performed within the simulation 
network and the various states and aspects of the traffic 
were recorded in real-time through the data collection 
processes within MACS. Questionnaires were taken by 
participants after each run followed by a post-simulation 
questionnaire once the final run was complete. A post- 
simulation debrief discussion was also conducted with the 
test and confederate participants where a variety of topics 
regarding the concept and the study as it was run were 
covered. 

5. RESULTS 

While data were collected and a number of metrics were 
analyzed for each of the conditions in this study, the 
following results section and ensuing discussion will focus 
on the medium (M) duration runs. This portion of the study 
was a 2 x 2 within subjects design where Traffic Density 
(NextGen Fevel A and B) was varied across Arrival Time 
Constraints (Baseline and STA). The metrics to be 
presented here will first be aircraft counts as they were 
recorded for each sector, followed by reported workload 
by the test participants, number of conflicts and conflict 
resolution data, separation violations, and finally 
subjective feedback from the participants. 


Table 1 Allocation of functions between the automation and controller 


Automation 

Controller 

Detect Separation Conflicts 

Supervise the automation 

Resolve trajectory-based conflicts (if within tolerances) 

Resolve trajectory conflicts flagged by the automation 

Resolve all time-critical traffic conflicts 

Monitor and maintain schedule compliance 

Alert controller to urgent problems 

Place aircraft back on trajectory following automated 
tactical maneuvers 

Provide trajectory planning assistance 


Use datalink to communicate 
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5.1 Aircraft Count 

Data for aircraft count are shown as a means of providing 
the context for the remainder of the results. Fig. 7 shows 
the average aircraft count in each of the four test sectors 
plotted over time for the NextGen Level A and B traffic 
densities. 



For aircraft count in NextGen Level A, the lower of the 
two densities, mean aircraft count in ZKC90 for the entire 
run was 34.69 aircraft in sector ( SD = 4.51), followed by 
ZID81 with a mean number of 25.94 aircraft {SD = 5.29) 
in sector. Sector ZKC 98 had a mean number of 20.86 
aircraft {SD = 2.20), followed by ZID80, which had the 
lowest mean aircraft count with a mean number of 13.91 
aircraft {SD = 3.71). Sectors ZKC90 and ZID81 were the 
largest in terms of airspace volume of the four. 

The NextGen Level B traffic density, found on the right 
hand side of Fig. 7, had higher mean levels of aircraft 
count in each of the sectors relative to that experienced in 
Level A. While the actual counts differed between the 
conditions, the order of sectors in terms of aircraft count 
held constant across the Level B conditions with ZKC90 
clearly having the highest mean aircraft counts with 45.76 
aircraft over time {SD = 5.90), followed by ZID81 with a 
mean of 34.66 aircraft {SD = 5.62), ZKC98 with a mean of 
28.00 aircraft, {SD = 4.10), and ZID80 with a mean of 
20.34 aircraft {SD = 4.75). 

5.2 Workload 

Throughout each run, workload prompts were displayed to 
the participants at three minute intervals through a 
workload keypad integrated into their display. The 
prompts consisted of the keypad highlighting at the 
appropriate time accompanied by an audible tone. The 
ratings were on an interval scale from one to six, with one 
translating to the lowest level of workload and six the 
highest. The analysis of workload was performed across 
Traffic Density Levels (A and B) and Arrival Time 
Constraints (Baseline and STA). Fig. 8 presents the mean 
workload ratings for each of these conditions. 

The descriptive statistics show that for Baseline-Level A, 
the mean reported workload was 1.78 {SD = 0.64) and the 
corresponding STA-Level A was slightly higher at 1.83 



Level A Level B 


Traffic Level 


Figure 8 Mean reported workload per condition (+/- 1 standard error) 

{SD = 0.74). For Traffic Density NextGen Level B, the 
Baseline-Level B condition had a mean reported workload 
rating of 2.01 {SD = 0.79) and the STA-Level B had a 
more noticeable increase in mean reported workload rating 
with 2.34 {SD = 0.79). 

To investigate the differences in mean reported workload 
ratings, a one-way, repeated measures Analysis of 
Variance (ANOVA) test was conducted first to test the 
difference between the Baseline and STA Arrival Time 
Constraints conditions. While Fig. 8 and the descriptive 
statistics showed the STA conditions being reported as 
generally higher in workload than the Baseline conditions, 
results from this test did not reveal any significant 
difference {F{ 1, 31)= 2.60, p> .05). An additional test was 
performed to examine the differences between the two 
Traffic Density levels in mean workload ratings. This test 
did yield significant differences where mean workload 
reported in Traffic Density Level B was significantly 
higher than in Level A {F{ 1, 31)= 14.85, p< .01). 

Also of interest in the analysis of reported workload was 
how it may have differed between sectors considering that 
the traffic complexities and characteristics differed 
between them. Fig. 9 shows how the overall workload 
differed between the sectors with ZKC 90 having the 
highest mean workload at 2.56 {SD = 0.50) followed by 
ZKC 98 at 2.09 {SD = 0.49), ZID 81 at 1.80 {SD = 0.33), 
and ZID 80 showing the lowest mean workload at 1.49 
{SD = 0.49). To investigate these differences further, a one 
way ANOVA was conducted where a significant difference 
in mean reported workload between the sectors was found 
{F{3, 60)= 7.65, p< .01). As a follow-up, Tukey’s Honestly 
Significant Difference (HSD) tests were conducted where 
it was found that ZKC90 had significantly higher workload 
than ZID81 {p< .01) and ZID80 {p< .01). The only sector 
that ZKC90 did not significantly differ from was ZKC98. 
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Figure 9 Mean overall workload per sector (+/- 1 standard error) 


5.3 Conflict Data 

To better understand the environment and operations as 
they played out in this study, the number of conflicts was a 
point of interest for analysis particularly as it lies at the 
heart of separation assurance. 


5.3. 1 Number of conflicts 

Fig. 10 shows the mean number of conflicts recorded that 
were predicted to lose separation within any of the four test 
sectors. From this figure one can see two visible trends: 
The STA conditions resulted in fewer conflicts in the test 
area regardless of traffic density, and overall the Level B 
density had a greater number of conflicts than Level A. 
Descriptive statistics show that the mean number of 
conflicts for Baseline-Level A were 68.75 (SD = 4.19) and 
lower for STA-Level A with 58.00 ( SD = 2.83). Baseline- 
Level B had a large increase with a mean number of 
113.00 conflicts (SD = 3.74). STA-Level B showed a 
fewer number of conflicts with a mean of 101.75 (SD = 
6.99). 



Figure 10 Mean number of conflicts predicted to lose separation within 
the test area (+/- 1 standard error) 


A one-way, repeated measures ANOVA was first run to 
examine the differences in the mean number of conflicts 
between the Baseline and STA conditions where it was 
found that the STA conditions did indeed result in 
significantly fewer conflicts (F( 1,7)= 30.52, p< .01). The 
differences between traffic density levels were also 


examined where it was also shown that Traffic Density 
Level A resulted in significantly fewer numbers of 
conflicts than in the higher density Level B (F( 1,7)= 
492.80, fX .01). 

5.3.2 Conflict resolutions 

In addition to the number of conflicts, an understanding of 
how the conflicts were resolved or attempted to be 
resolved was desired. This involved an examination of all 
of the clearances sent to aircraft in conflict. There were 
three types of resolutions that could be sent via data comm: 
strategic automated resolution (Automated Resolution), 
uplink by controller of resolutions deferred by the 
automation (ATC Resolution), and the tactical TSAFE 
maneuvers sent to aircraft with two minutes or less to loss 
of separation (TSAFE). 

Fig. 11 presents these data for each of the conditions where 
naturally the overall number of resolution clearances sent 
resembles the conflict numbers shown in Fig. 10. 

From Fig. 1 1 one can see that, as designed, the majority of 
conflicts were resolved by the strategic conflict resolution 
automation. Upon further inspection of the strategic 
resolutions, increases in the contributions of ATC 
resolutions to the overall number were found as the traffic 
density level increased. In Baseline-Level A, ATC 
resolutions accounted for 14% of strategic resolutions. 
This increased to 21% of strategic resolutions in Level B. 
This was likewise the case for the STA conditions where at 
the Level A density ATC resolutions accounted for 20% of 
strategic resolutions and increased to 25% in Level B. 

A similar trend is also observable when accounting for 
TSAFE resolutions and their contribution to the overall 
number of clearances. For the Baseline conditions, there is 
an increase in TSAFE’s contribution from 12% to 22% as 
traffic density increases from Level A to B. This is also the 
case for the STA condition where there is an increase from 
16% to 24% from Level A to B. 


■ TSAFE ■ Automated Resolution ■ ATC Resolution 


I ■ I i 

Baseline STA Baseline STA 

Level A Level B 


Figure 1 1 Mean numbers of Automated, ATC, and TSAFE resolutions 
per condition 


To better understand the composition of the clearance 
numbers in terms of each sector’s contribution, the overall 
number of clearances was separated out accordingly. These 
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mean numbers reflect an overall view of resolutions sent 
throughout the study, regardless of condition. 


Fig. 12 presents the mean number of all conflict 
resolutions sent per sector. The most striking result from 
this portion of the analysis is sector ZKC 98’ s share of 
conflict resolutions relative to the other sectors. For 
example, ZKC 98 had a mean number of 111.75 total 
resolutions sent while ZID 81 sent a mean total of 86.00 
resolutions. ZKC98 had required a greater number of 
resolutions sent despite the fact that it had a lower level of 
traffic (see Fig. 7). This is due in part to the smaller size of 
ZKC 98 but, more importantly, also to the complexity of 
the sector. A more striking example of how the complexity 
of ZKC 98 affected the composition of clearances sent is a 
comparison of the mean number of TSAFE resolutions. All 
things being equal, one might expect each of the four 
sectors to be responsible for 25% of the total number of 
TSAFE resolutions sent. However, as shown in Fig. 12, 
this was not the case. Of the total mean number of TSAFE 
resolutions sent, ZKC 98 accounted for 55% whereas the 
other sectors ranged between 12% and 17%. The 
percentage for ZKC 98 was a direct result of its complexity. 
The implications this may have had on workload and 
acceptability will follow in the Discussion section. 



Figure 12 Mean numbers of Automated, ATC, and TSAFE resolutions 
per test sector 


place. From this figure one can see that the Level A traffic 
density resulted in one OE each for the Baseline and STA 
conditions. This number increased as traffic density 
increased to Level B where both Baseline and STA 
conditions resulted in a total of six OEs each with four and 
two additional PEs respectively. 



Baseline STA Baseline STA 

Level A Level B 


Figure 13 Total numbers of Operational Errors and Proximity Events per 
condition 

As shown in Fig. 14, examining the number of separation 
events by sector rather than condition revealed that ZKC98 
had the highest number of OEs with seven followed by 
ZKC90 with five and ZID80 with three; ZID81 had none. 
In terms of PEs, however, ZID80 had the highest number 
with three while the other sectors each had one. 


■ Operational Errors ■ Proximity Events 
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5.4 Losses of Separation 

The benchmark for safety in any air traffic environment, 
excluding mid-air collisions, is loss of separation events. 
These events can be broken down according to two 
classifications: Proximity Events (PE) and Operational 
Errors (OE). The PE classification, the less severe of the 
two, refers to events where a pair of aircraft has a closest 
point of approach (CPA) that is between 8.33 km and 9.26 
km (4.5 and 5.0 nautical miles (nm) respectively) laterally 
and 243.84 m (800 ft) vertically. Operational Errors 
involve aircraft pairs that have a CPA of less than 8.33 km 
(4.5 nm) laterally and 243.84 m (800 ft) vertically. 

For this part of the study, there was a combined total of 20 
PEs and OEs. Fig. 13 presents the breakdown of these 
numbers according to the conditions in which they took 


Figure 14 Total numbers of Operational Errors and Proximity Events per 
test sector 

5.5 Subjective Feedback 

Following each run and at the conclusion of data collection 
questionnaires were distributed to the participants in order 
to elicit their subjective feedback on a range of topics. 
References [12] and [14] provide additional data from this 
feedback. This section will focus on a subset of the 
responses as they relate to situation awareness, 
acceptability, and overall impressions of the concept. 

5.5. 1 Situation A ware ness 

The concept as it was implemented and the environment in 
which it was introduced was a significant departure from 
what the participants experience today. Because of that, the 
impact of operations on their situational awareness was of 
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interest. To gain insight into the impact, the Situation 
Awareness Rating Scale (SART) [15] was used after each 
run where participants were asked to give responses to 
three questions on their understanding of the situation, 
demand that it placed on them, and their supply of 
attention capacity during the run. Participants answered the 
three SART questions on a 7-point scale from 1= “very 
low” to 7 = “very high”. The responses to these questions 
are then converted into an overall rating of situation 
awareness. 

All three questions were answered using the full range of 
the scale - from 1 to 7- with a good spread of answers 
particularly between the sectors. As shown in Fig. 15, 
answers for the understanding question tended to be at the 
higher end of the scale with few lower (“poor 
understanding”) responses. Overall, participants felt they 
had a “very good understanding” of the situation (M=6.33, 
SD=1.37), an “average” supply of attention capacity 
(M=4.25, SD = 1.89), and that there was only “a little” 
demand on their attention (M=2.96, SD = 1.37) (see Fig. 
15). These responses combine to indicate that participants 
felt they had a “reasonable situation awareness” (M=5.08, 
SD=2.28). 



Figure 15 Mean SART ratings compared by the level of traffic density 

5.5.2 Acceptability of Operations 

Subjective feedback was also sought regarding the 
acceptability of operations as they were experienced in this 
concept’s environment. This was gained through the 
presentation of questions closely aligned with the 
Controller Acceptance Rating Scale (CARS) [16]. 
Participant answers are compiled on a scale from one to 
ten where one indicates that the operation is not safe and 
ten indicates the system is acceptable. Participants rated 
the separation assurance operation as safe less often under 
NextGen Traffic Level B (67.5% of the time) than under 
Level A (90.6% of the time). A general analysis of the 
comments indicated that it was not the absolute volume of 
traffic that concerned participants but that the situations 
became more complex as the traffic increased (and finding 
a conflict solution with a clear path became less and less 
easy). For example, one participant commented that, 


“Traffic level really not that much of a factor. Number of 
conflictions at any one moment were more of a factor.” 

5.5.3 Overall Concept Impressions 

Following the completion of the data collection phase of 
the study, a post-simulation debrief and discussion was 
conducted. This allowed both the researchers and 
participants to converse openly and freely regarding their 
ideas and opinions on the concepts presented and how they 
were carried out in the study. With respect to the concept, 
participants provided positive feedback with quotes such 
as, “you’re on the right track,” and, “it seems fairly natural, 
why not do it?” However, some concern was expressed 
regarding some of the more technical aspects of how the 
concept could be realized. One participant remarked, “...it 
seemed as if controller and automation fought against each 
other at times to resolve conflicts.” This was mainly in 
response to the hands-off approach taken in the operation 
of automated tactical conflict resolutions. Despite these 
types of concerns, however, the overall mood and 
assessment of the concept was positive and can best be 
summed up by one of the participants who said, “It’s 
inevitable, I think the concept is strong, it needs work and 
testing, I think it’s the way we’re going to go.” 

6. DISCUSSION AND CONCLUSIONS 

The results presented in this paper were from a larger study 
that examined the functional allocation of ground-based 
automated separation assurance, which was also part of an 
overall study that examined the functional allocation of 
separation assurance between the flight deck and a ground- 
based ATC system. These studies were part of a continuing 
series of investigations into the realm of automated 
separation assurance and its related issues particularly as 
they relate to human-systems integration. 

The results presented here were based on the medium 
duration runs of the larger study that were 30 minutes in 
length and varied Arrival Time Constraints (Baseline and 
STA) across two levels of Traffic Density (NextGen Level 
A and B). With the exception of number of conflicts where 
the STA condition had significantly fewer conflicts than 
Baseline, comparisons between Baseline and STA did not 
yield many insightful results. The most noteworthy 
comparisons were between the Traffic Density levels. 

In Fig. 7 the plots of sector counts over time for each of 
the test sectors showed that NextGen Level B had 
generally higher sector counts but that the differences 
ranged between the sectors. These plots provided an 
important context in which to view the subsequent results 
because what was witnessed was a shift away from sector 
count alone being the driving factor for workload and 
safety to other factors such as local complexities within a 
sector. 

For example, traditionally one might expect workload 
ratings to be highest for the sectors with the highest sector 


10 


ENRI Int. Workshop on ATM/CNS. Tokyo, Japan. (EIWAC 2010) 


counts, which were, in decreasing order, ZKC90 and 
ZID81, the two largest sectors. However, an examination 
of reported workload revealed that it was ZKC98 that 
resulted in higher workload than ZID81, despite lower 
levels of traffic. It should be mentioned that although there 
were inter-sector differences, even the highest overall 
mean workload ratings were still relatively low and within 
an acceptable range. 

In this simulation, ZKC98 was at once one of the smallest 
sectors while also being the most complex. This 
complexity existed by virtue of heavy arrival and departure 
corridors within the sector where transitioning aircraft 
interacted with aircraft at level flight. It was this type of 
interaction and inherent complexity that allowed for the 
observation of the equal or greater impact of complexity 
over that of simple sector count on operations and 
acceptability. 

This was first evident through the examination of conflict 
data where it was initially found that as traffic density 
increased from Level A to B, there was a significant 
increase in the number of conflicts. This naturally resulted 
in a greater number of conflict resolutions to be issued. 
Upon further examination of clearances by sector it was 
found that despite ZKC98 having the third lowest level of 
traffic count, it had the highest number of resolutions sent. 
This was particularly the case in terms of tactical 
resolutions where more than half of all sent were in this 
sector. 

For sectors ZKC98 and ZKC90, the number of operational 
errors followed the same trend as the number of clearances 
sent respectively. This was not the case, however, with 
ZID81, which had the second highest levels of traffic and 
the third highest number of clearances sent but did not 
have any resulting operational errors. 

ZID81 had the second highest levels of traffic but was 
managed such that operations and workload were tenable 
with the least impact on safety with only one proximity 
event. This speaks to the potential that greater levels of 
traffic can be safely accommodated but also that particular 
attention needs to be paid to the complexity of the local 
environment and the ability to mitigate or manage it 
effectively. 

While the ultimate goal of any system is to eliminate 
separation violation occurrences, they did occur here and 
are being investigated. Preliminary results show that late 
detection and the difficulties related to accurate trajectory 
predictions for transitioning aircraft played significant 
roles in the violation of separation minima. Despite the 
complexities described and the number of operational 
errors, while attempting to assess the success rate of the 
conflict resolutions sent it was found that over 99% of 
conflicts were successfully resolved and avoided an 
operational error in this portion of the study. 


Subjective feedback from the participants provided this 
research effort with a different perspective on operations 
apart from the objective data. Situation awareness was 
rated as being “reasonable” even with the levels of 
automation in operation but that it was not the same for all 
sectors. In terms of acceptability, participants rated 
operations at the lower traffic density quite highly. 
However, this showed a reversal as the density increased 
such that there was a 23% decrease in participant ratings. 
This did not seem to affect their overall assessment of the 
concept, however, where the general consensus was that 
the approach taken was a viable way forward. 

To conclude, in this study it was found that the increase in 
traffic density resulted in the greatest impact on operations. 
However, it was not aircraft count alone but local 
complexities that affected the observed results. This 
observation coupled with feedback provided by the 
participants speaks to the promise that this functional 
allocation concept of ground-based separation assurance 
holds for the ability of the NAS to accommodate the 
environment envisioned as part of NextGen. 
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